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Introduction 

The Appendix to the Data Quality Report Series is 
set aside for more detailed descriptions of the quality 
control and data evaluation techniques used to reach the 
conclusions reported in the other sections. Included 
here are definition of terms, directions to the analyst 
with respect to current procedures, background material, 
rationale, technical details etc. It will be prepared 
and released in several sections covering specific sub- 
jects, and can be considered the equivalent of a Labora- 
tory Services Branch Quality Control Handbook. 

This first section covers topics related to the 
monitoring, control and evaluation of analytical repro- 
ducibility by means of standard deviation calculations. 
The difference between random deviation and systematic 
error and their affect on bias is covered. Definitions 
of terms used (such as precision, detection criteria, 
sensitivity, resolution) are provided. 
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PART A: 



DEFINITIONS 

Reproducibility in reporting results is generally 
limited by one of the following factors in order of 
increasing significance 

a) electronic (instrument) noise observed under 
extreme range expansion. 

b) inability to observe small changes in reading 
because of insufficient range expansion. 

c) analytical noise, eg. non reproducible colour 
development or noisy reagent background y 
independent of sample preparation procedures. 

d) sample preparation noise, eg. poor reproduc- 
ibility after sample digestion. 

e) sample non-homogeneity. 

f) poor sampling or sample handling and preserva- 
tion. 

Field and/or time variability do not relate in any 
way to precision of analysis. If a representative 
sample is difficult to obtain then one should recognize 
that a single sample or even the average of several 
samples may not adequately describe the object of 
s tudy . 

Quality control and performance evaluation attempt 
to minimize and quantify the data scatter resulting from 
the measurement process . In order to discuss these 
topics it is necessary to appreciate the intent of 
certain technical terms which are described below. The 
factors 1.96 and 1.645 which occur in some of the 
definitions are taken from the Cumulative Normal 
Frequency Distribution Table available in any book on 
statistical methods and are used to predict the pro- 
bability that a large deviation from the average will 
occur. The t-distribution which allows broader limits 
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when only a few datum are available is not considered 
appropriate here, although the standard deviations are 
usually calculated from a small amount of data, because 
the analyst's confidence in, and knowledge of, the 
'ruggedness' of his methods compensates for the statis- 
tical lack of confidence . 

Deviation ; Unavoidable random fluctuation in measure- 
ments caused by insensitivity to small changes 
and difficulties inherent to any measurement 
process. 

Error : Gross deviations, beyond those normally 
expected, caused in a single instance by 
misuse or misapplication of the measurement 
system. 

Systematic Error : Reproducible correctible deviation, 

usually resulting from improper calibration of 
the measuring system against some reference 
system. Usually this error will affect all 
data in a given set of measurements but will 
vary from one calibration to the next because 
of random deviations inherent to the calibra- 
tion process. 

Bias: If the reference used to calibrate the 

measuring system is correct, then, on average, 
systematic errors from calibration to cali- 
bration will cancel out. If this is not true, 
then, on average, the results are biased. 

When estimating the average value of something, 
precision is improved by increasing the number of measure- 
ments , thereby increasing the confidence one has that the 
overall average will be reproducible. Even if single 
measurements are imprecise the average over many measure- 
ments can be made as precise as necessary. Repeated 
measurement will not eliminate bias i.e. improve accuracy 
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Resolution ; smallest change in instrument response 
which is discerned under routine reading 
conditions . 

Sensitivity : concentration or weight change equivalent 
to the resolution. 

Standard Deviation : is a measure of reproducibility 
that would be obtained if a sample were re- 
analyzed. It can be determined in one of two 
ways 

a) a single sample is analyzed n times to 

obtain results Xi , X 2 , X . The 

average value is calculated X = ZX/n. 

The deviations (X - X) are used to 

n 

calculate the standard deviation 

s = / Z(X n - X) 2 /(n - 1) 

b) n samples are analyzed each in duplicate 
to obtain differences between results 
(Xj - X 2 ) 1 (Xi - x 2 ) n . 

These differences are used to calculate 
the standard deviation 



s = / Z(Xi - X 2 ) 2 /2n 

n 

Both of these estimates of s are equally valid and 
can be used to predict the likelihood of finding a 
large difference between any two results , providing 
they were obtained under similar circumstances. On 
average, a single result will be within i s of its 
average value in two out of three cases. It will be 
within + 1.96 s of the average in 19 out of 20 times, 
i.e. 95% of the time . Therefore the difference between 
any two single results will be within ± 1.96 /2s 95% of 
the time, where the /2 factor accounts for the varia- 
bility in both results. 



I A- 4 

Precision : A statement of the range within which random 
deviations can usually be expected to fall. 

Within-run Precision : A statement of the range of 

deviation expected if measuring system is not 
changed. 

Between-run Precision : A statement of the range of 
deviation expected if the calibration is 
changed between readings. 

While within-run precision is independent of cali- 
bration, the between-run precision will depend upon the 
amount of control exerted over the calibration process. 

Detection Criteria : the amount of analyte required to 
be found to ensure that when it is 'absent' 
it will not be reported as 'present'. 

Detection Limit : the amount of analyte required to be 
present to ensure that when it is present it 
will not be reported as absent (i.e. less 
than detection criteria) . 

The terms 'precision', 'detection criteria' and 
■detection limit' in the strictest sense should include 
a statement of the confidence with which the statement 
is likely to be true, eg. 9 5% or 99% level of confidence. 
The most usual case is to use 95% confidence levels. 

Precision (95%) : implies that the difference between 

two results will not exceed 1.96 /2~ s « 2 .77 s 
more than 5% of the time (°= = 0.05). 

Detection Criteria (95%) : implies that if the analyte 
is absent, a positive result greater than 
1.645 £ (where s is determined for 

data close to zero) will be obtained less than 
5% of the time (« = 0.05). 
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NOTE: if the analyte concentration is equal to the 
detection criteria concentration then 50% 
of the observed results will be less than d.c. 
Thus 50% of the time the analyte will be 
incorrectly reported as absent. ($ = 0.5) 

Detection Limit (95%) : ensures that when the analyte 

is present it will be reported as absent less 
than 5% of the time (3 = 0.05) if the level 
of analyte is at least 2(1.645)s = 3-29 s . 

Accuracy : a statement of the amount of bias which has 

been found, measured relative to some absolute 
reference. If precision is sufficiently poor, 
bias will be difficult to detect or correct 
however its influence on averaged data will 
still be real and significant, and its 
presence when known should be noted. 

The standard deviations used in the above definitions 
should be appropriate to the concentration level being 
reported. They should be based on reproducibility of 
results, not readings. In automated systems a sample 
can be analyzed sequentially several times. Each peak 
should be read against an independent estimate of the 
baseline to obtain individual independent results and 
should be spread through the run, not together. The 
practice of basing standard deviation estimates on 
replicated readings is inappropriate to the intended 
use. Such estimates will be low by at least /2 because 
no allowance is made for variability in the choice of 
baseline or blank. 

In most cases the standard deviation based on with- 
in run duplicate analysis of routine samples of the type 
of interest should be used in calculating the above 
confidence levels. 
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NOTE ; DETECTION LIMIT, DETECTION CRITERIA AND PRECISION 
STATEMENTS ARE ADVISORY IN NATURE. FIELD AND/OR 
TIME VARIABILITY USUALLY DO NOT PERMIT APPLICATION 
OF THESE CONCEPTS TO ASSIST THE DATA USER IN 
INTERPRETING HIS RESULTS UNLESS IN FACT ANALYTICAL 
EFFECTS ARE LIMITING. 

NOTE : THE DETECTION CRITERIA OR DETECTION LIMIT ARE NOT 
TO BE USED TO LIMIT THE REPORTING OF RESULTS WHEN 
A NUMBER IS AVAILABLE. IF DESIRED THE LOWER 
RESULTS CAN BE REPORTED IN BRACKETS OR WITH A 
CODE TO INDICATE LACK OF CONFIDENCE. 

NOTE ; THE SYMBOL < OR AN APPROPRIATE CODE SHOULD BE 

RESERVED TO INDICATE THAT NO NUMERICAL ESTIMATE 
WAS POSSIBLE AND THAT THE BEST ESTIMATE IS 
POSSIBLY LOWER THAN THAT REPORTED. 

NOTE ; DATA REPORTED AS LESS THAN ( < ) SHOULD BE INCLUDED 
IN AVERAGING AS IF THEY WERE ZEROS. THE < SYMBOL 
OR ITS EQUIVALENT CODE SHOULD NOT BE LOST WHEN 
STORING OR RETRIEVING DATA. 

NOTE ; IF THE DATA USER REQUIRES BETTER PRECISION OR 

LOWER DETECTION LIMITS HE SHOULD REQUEST WHETHER 
THE ANALYTICAL SYSTEM CAN BE ADAPTED TO HIS NEEDS. 
IF NOT, REPLICATE SAMPLINGS OR ANALYSES MAY BE 
NEEDED SINCE THE STANDARD DEVIATION OF AN AVERAGE 
IS 1//H BETTER THAN THE STANDARD DEVIATION OF A 
SINGLE RESULT. THIS APPROACH IS NOT VERY EFFI- 
CIENT HOWEVER BECAUSE 9 REPLICATE ANALYSES WILL 
IMPROVE PRECISION AND DETECTION CRITERIA BY ONLY 
3X. 



APPENDIX SECTION It ANALYTICAL REPRODUCIBILITY 
PART B: 



DUPLICATE ANALYSIS: A MEASURE OF IN-RUN ANALYTICAL 
PERFORMANCE 



It is generally agreed that regular duplicate 
analysis of at least 1 in 20 routine samples is required 
to monitor adequately the level of repeatability being 
maintained under routine conditions. The question re- 
mains as to what should be done with this data since if 
it is not used in a regularly defined fashion the add- 
itional work required is difficult to justify. Certainly 
it is not sufficient to allow this data to accumulate 
unreviewed. Data derived in this way can be used for the 
following purposes; 

a) To determine that the precision of analysis for 
today is within acceptable limits. 

b) To document the distribution of difference 
between duplicates versus the operating range. 

c) To document the distribution of samples versus 
concentration range to assist in evaluating 
the suitability of the operating range. 

d) To calculate the within-run standard deviation 
for the analytical process at various levels of 
concentration under routine conditions. 

The minimum required must be to sort this data, as 
it is generated , according to sample type, concentration 
level, and difference between duplicates. This is read- 
ily achieved by use of the form shown on the next page. 
Each square is used to accumulate the number of samples 
observed to fall within the specified concentration 
range and for which the difference between duplicates is 
found to be as shown. A dot represents one sample and a 
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bar represents five samples. Thus, as data becomes 
available, points are added to the appropriate square 
on the form. 

The concentration range axis is divided into 2 
parts, covering the operating range of the instrument 
in steps of 5%. This permits an assessment of the effect 
of concentration on standard deviation, and also reflects 
the distribution of samples concentrations. Off scale 
concentrations can be handled in one of two ways : 

a) plot the final result obtained on a separate 
form for the sample type and concentration 
range covered by the dilution factor used, or 

b) plot the on-scale readings obtained, before 
applying dilution (or concentration) factors, 
on the same form used for the regular on-scale 
samples . 

The latter procedure is easier in terms of the 
number of separate forms needed, and, unless samples are 
included which required an exorbitant dilution factor, 
reflecting a probable change in character of the sample, 
it is usually found that the difference between dupli- 
cates encountered for readings (as opposed to results) 
is relatively independent of dilution factor. 

In order to simplify the calculations scaled differ- 
ences have been used in the table to provide D 2 values. 
The actual concentration difference equivalent to the 
scaled differences would be entered on the left side of 
the diagram. Thus if the factor f=0.002 mg/1 were used, 
the difference scale spaces would be labelled 0.000, 
0.002, 0.004, 0.006, etc., corresponding to D=0 , 1, 2, 
3, etc. respectively. In general the f factor should be 
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chosen such as 0.001, 0.002, 0.005, 0.01, 0.02, etc., 
so as to be smaller than the expected standard deviation, 
or equal to the smallest reading interval, (eg. if read- 
ing interval is 0.01 and standard deviation is 0.035 
choose f=0.02 or 0.01 mg/1) . 

Note that differences between duplicates are ex- 
pected to occur. If none are observed steps should be 
taken to read results so as to record the next signifi- 
cant figure. 

When sufficient data has been accumulated in this 
fashion, the procedure outlined below (refer to example 
on opposite page) should be followed. 

1) Count the number of samples for which a part- 
icular difference was observed, for each of 
the three concentration ranges (0-20%, 20-50%, 
and 50-100% of full scale) and enter these 
numbers (n) in the corresponding line of the 
table. 

2) Multiply n x D 2 and enter this result in the 
appropriate space in the table. (Note scaled 
values of D 2 are already written in the table) . 

3) Sum the columns in the table to obtain £n and 
ZnD 2 for each of the three concentration ranges. 

2 

4) Calculate the scaled variance V= *s(EnD /£n) and 

enter this result in the appropriate space on 
the table . 

5) Calculate the standard deviation by taking the 
square root of the scaled variance and multi- 
plying this by the concentration/unit D factor 
f . 




Ontario 



.- 

{J 



p 

c 
o 

CD 

S 

OJ 

JQ 

QJ 

U 

c 
c 
u 
a> 
in 

•H 

Q 



Ministry ol the 
Environmenl 



Concentration as a percent of full scale 
20 50 



TABLE FOR CALCULATING STANDARD DEVIATION 



100% 



i~- 







•v3 








<-* 


00 
Of 






r^ 


fO 


— 


o- 


zr- 


oo 


l/> 


h- 


-4? 


rn 


>• 


Scaled 
D 2 












































400 








































361 


.OlO 

.oiB 

. Olio 
. OI7_ 

.oio 
■ oofc 










































324 










































289 










































256 










































225 










































196 








































169 










































144 










































121 






























* 




; 


* 






100 


















m 
























81 
















* 
• 




• 


:. 




* 
















64 


* 










* 


* 


* 


* 
























49 






m 




• 
• 


• 




* 
1 
■ 


•< 

* 


• 
* 


t 


• 


• 
















36 


t 
■ 




• 


• 


» 


• 
• 




. 


• 








• 


• 


« 


« 


• 
• 
• 


• 
t 

* . 


* 


* 


25 


• 




* 


• 
• 
• 


1 
1 


• 


• 


• 
• 
* 


1. 


• 
• 
• 


■ 
• 


* 


















16 


• 
• 
• 

m 


: 


• 
• 


• 
* 
* 


| 
■ 
• 


* 
• 
• 


1 


• 
• 


« 
• 


• 






















9 


od4- 


I 

II: 
1: 


* 
1 

ir 


1 


• 
• 


• 


■ 
• 


i 


• 
• 


1. 


4 

* 

• 


• 
* 


* 
• 
* 


* 
• 














4 


.o OX 


1: 
1: 


1 
I. 


1 

1: 


I. 


• 
a 
4 

• 


1. 


■ 
1 


• 
















— 






1 


O-OO 


1 


1. 


I 
• 
• 
« 


1 


■ 
• 


1 


* 
* 


■ 


* 
* 


* 

* 
• 


• 
* 
• 


* 


a 


• 
— 


1 ° ' 
































I • 







o - ?"* is 
n nD 2 



o 




* 





Q 



+ 

o 



o 



00 

o 



r4 



6« 



Concentration units 



V = EnD z /2En 
s = f/V 




O.ooSX 



20 - 


50% fs 


n 


nD 2 


— 




— 




— - 




•— 




m """ 







o 


1 


%\ 


V 


2<St> 


JL 


l^k 


12- 


432- 


7 


n* 


2o 


320 


JO 


ISO 


^1 


too 


£L 


24 


3o 


0. 


ilL 


n&4 




£0% 




. 00S0 



SO - 100% fs 



n 


nD 2 






































O 


O 


J- 



90O 






32.0 



it 




•44 






24 


32- 



0. 


-70 


iZio 




133 




0. 007 z 



1 



IB-6 

6) Sum the number of samples occurring in each 
column of the diagram and enter the result at 
the top of the column to obtain the distribution 
of samples versus concentration range. 

Although this technique for recording the results of 
duplicate analysis is not intended to serve a quality 
control function on a day-to-day basis, it is obvious that 
as data is accumulated a pattern will appear which will 
serve to indicate that today's entries follow or deviate 
from the usual pattern. Excessive deviation will then 
suggest a breakdown in the routine performance. 
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PART C; 

DETECTION OF SYSTEMATIC ERROR IN TRACE ANALYSIS 

The need for control of systematic error within the 
analytical laboratory is well evidenced by the number of inter- 
laboratory comparisons which have shown by means of Youden's 
technique Cl'3, that it is the most significant source of 
deviation between laboratories. Many analysts, when queried 
directly, are unable to comment on the extent or even existence 
of systematic error in their laboratory. Since inter-comparisons 
are carried out infrequently, insufficient data is available to 
determine whether an observed error in the result reported by a 
given laboratory applies only to the run in which the result was 
obtained or whether it is typical of that laboratory's routine 
work. 

Systematic error can be separated from random 
deviation by the use of two similar control samples or standards. 
This technique was applied by Youden to inter laboratory compa- 
rison in which several laboratories analysed each of two control 
samples. We have applied the same technique to intralaboratory 
work by analyzing the two control samples once per analytical 
run over a period of 15 or more runs, and have been able to 
detect and correct significant systematic error as it occurs. 

Theory 

The object in controlling analytical performance 
is to ensure immediate detection of any deterioration in 

instrumentation, technique and/or standardization. Most 
observed deviations in analytical results can be considered 



IC-2 
insignificant when measured against routine performance. 

Unusually large deviations should then occur only rarely as 

long as the system produces results distributed normally 

about the mean value. The standard deviation 's' of the 

available data is commonly used as the measure of performance 

and is used to infer that deviations larger than 2s should 

occur less than 5% of the time. This inference is true if 

the data is 'Normally* distributed as is usually the case 

when it has been obtained under one set of conditions. If the 

data is provided from different analytical runs, errors made 

in determining either the blank or calibration slope will 

result in a broader non-normal distribution of data. However, 

there is usually insufficient data to demonstrate this, unless 

two samples run simultaneously show the same error. 

If the theoretical relationship between concentration 

C and instrument response R can be written 

C = C + kR (1 

o 

where C = concentration required for nil response 
k = linear slope relating C and R 

then the estimated concentration C for an observed response R 
will be given by the equation, 

C = (C + e ) + kR(l + e_) + c (2 

O O ^ r 

where e - systematic error in the blank 
o 

e. = systematic error in the slope 

c = random deviation 

r 

Thus the total deviation can be described by the formula 

c = (C-C) = c r + e 

= c+e+e.C (3 

r o k 
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At levels of analysis close to C Q (eg trace analysis) 
the most significant source of the systematic error e is in 
estimating the blank value C and e is independent of concentration. 
At higher levels e becomes increasingly proportional to concen- 
tration due to the term e., but since the random component c r is 
not necessarily proportional to concentration, the observed 

total deviation (c + e) need not increase linearly with 

r 

concentration. (For these reasons the common practice of 
reporting relative standard deviations, rather than absolute 
standard deviations, can be misleading) . 

The separation of systematic and random deviations 
is achieved in the following manner. When several analyses are 
performed on each of two control samples, A and B, the deviations 
a and b, observed for a given pair of results about their mean 
values A and B, can be separated into their random components 
a and b and their systematic error component e. Provided that 

A and B are close in concentration the error component e will 

be common to both A and B. Thus 

a = (A-A) = a + e (4a 

r 

b = (B-B) a b + e (4b 

(a+b) = (T-T) = a r + b r + 2e (4c 

(a-b) = (D-D) = a r - b r (4d 

where T = (A+B) and D = (A-B) 
Note that since the values a„ and b_ are each normally distri- 
buted and randomly associated (ie some pairs of a.^, h^ are at 
least partially self cancelling) , not all of the observed 
deviation common to A and B is due to systematic error. 
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By means of the usual formula for calculating 

variance namely: 

s 2 - I (A-A) 2 = Iai_ (5 

A (n-1) (n-1) 

it can be shown that 

s 2 = 1 Z(a 2 + e 2 + 2ea ) < 6a 

A TH-T7 ' r 

s 2 - i E(b a + e 2 + 2eb ) < 6b 

B Tn^ir • r 



s 2 „ ! E[a 2 + b 2 + 4e 2 + 4e(a +b ) + 2a b ] 

T Tz=ir - r 



s 2 = 1 2 (a r 2 + b 2 - 2a b) 

D Tn^TT r 

it can be shown that 

a) within-run variance s 2 = £(a + b r ) 



Tft=T} 

= )5(s D 2 ) + Za r b r /(n-l) 



Zei £e(a r + b r ) 



2 - 7o2 



(n-1) 



c) total variance s 2 = (s r + s e ) 



(6c 
(6d 



(7 



b) systematic variance s - &£ + / , . 

e Cn-1) (n-1) 

- ^ S T 2 " S A 2 - S B 2) " Za r b r (8 



(9 



- ^(s A 2 + s E 2 ) (10 



= ^(s T 2 + s d 2 ) (11 



Provided that there is sufficient data so that 

a . b and e can exert their random association, the cross- 

r r 
product terms will be negligible. 

How&Mcr,the cross-product terra in equations 7 and 8 
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cannot be determined so that for insufficient data the within-run 
variance will be over estimated by an indeterminate amount. 



Application 

The primary purpose of the two-sample control procedure 

is to flag those analytical runs which may result in significant 

systematic error. The control technique involves recording the 

values of A and B obtained for each of the control samples, 

calculating their sum T and difference D, and plotting these 

values in time sequence. If no systematic error is present 

then s will equal s but in general this is not found to be true. 
T D 

Warning limits of + 2s D and control limits of ± 3s D can be used 
to define outliers. Outliers in the T plot confirm the presence 
of significant systematic error whereas outliers in the D plot 
indicate a significant level of random deviation. In practice 
control limits are based on a previous set of control data. 
Care must be taken not to set control limits tighter than those 
the system is routinely capable of maintaining. 

It is also useful to maintain a sequence plot of 
B vs A (Youden plot) in order to clarify the long-term 
performance of the analytical system. In the absence of systematic 
error, the lines joining sequential A, B points will be randomly 
oriented. As systematic error increases these lines will tend 
to be oriented from upper right to lower left quadrants and the 
points will change from a circular into an elliptical distri- 
bution along this axis, provided only that variance of A and B 
are approximately equal. 
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Any reasonably stable material can be used to prepare 
the two control samples. Sufficient material or sample is 
required to provide for at least 20 to 30 analyses over a 
period of one to two months. As few as 10 sets of analyses have 
proved useful however. Both artificial and natural samples 
have been used with success, depending upon whether control of 
instrumentation only, or control of the entire analytical 
process was required. 

The choice of concentrations to be used for A and B 
control samples or standards depends upon the relative signi- 
ficance of e versus e, type systematic error. Two controls 
at the upper end of the operating range will control slope 
error and will be relatively unaffected by blank error, whereas 
two low-level controls will monitor systematic error in the 
blank. Since most trace analysis is performed at the low end 
of the operating range, the control of e Q error is critical if 
small changes in the environmental levels are to be detected. 
Notice that the actual strength of the check samples need not 
be known in order to apply this control technique so that 
stabilized field samples can be used to provide control at the 
actual level of interest. 

Figure 1 demonstrates the use of this technique 
to document the effect of systematic error in the taring of 
50 ml porcelain dishes (weighing about 40 grams) , for use in 
the gravimetric determination of Total Dissolved Solids. Each 
day two tared dishes arbitrarily assigned A and B were filled 
with 50.0 ml of distilled water and dried overnight at 103 °C 
along with the routine samples being analyzed. After cooling 
in a desiccator the next morning the dishes were reweighed and 
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Figure 1 50 ml distilled water blanks, dried overnight 
at 103 °C in pretared 50 ml porcelain dishs 
arbitrarily labelled A and B, to demonstrate 
use of T, D and Youden control charts. 



I 
I 



IC-8 

the 'blank' values were calculated. Although most results 
fell in the range + 10 mg/1 some values were as high as 28 mg/1. 
Examination of the T and D and B vs A plots points up the signi- 
ficance of systematic error which results in a between-run 
precision of 10 mg/1 (2^ times the within-run precision) as shown 
in Table 1 and 2. Because of this, the use of Specific 
Conductance has been adopted, for natural water samples containing 
less than 400 micros iemens/cm, as a more precise measure of 
dissolved solids in this type of sample. 

Figure 2 documents systematic error in the analysis 
of Sulphur in vegetative material by X-ray fluorescence. In 
this case the A and B samples were selected to be at about 15% 
and 45% of full scale. Pellets were prepared from each sample 
and were reanalysed once per run until they started to disinte- 
grate, at which point fresh pellets were prepared from the 
same two samples. The data was accumulated over a period of 
six months. It is obvious from figure 2 that systematic error 
is occurring but since the standard deviation of the data is 
proportional to concentration, direct analysis of the raw A and B 
data is invalid. (This is not a common observation for most 
of the procedures run in our laboratories) . 

In order to investigate this system the B data was 
therefore arbitrarily divided by three, and in addition the data 
was split into two portions covering periods before and after 
a major instrument overhaul. The results of this treatment 
are shown in figures 3 and 4 and tabulated in tables 1 and 2. 
Examination of the B vs A plots demonstrates that the overhaul 
was successful in reducing sources of random deviation but that 
systematic error was not eliminated. 
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Figure 2 % sulphur in vegetation by XRF analysis, over a 
period of six months. Equipment overhauled at 
time indicated by arrow. Note variation in 
A not equal to variation of B. 
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Figure 3 Replot of XRF data prior to overhaul of the 
instrument (B values are 1/3 those shown in 
figure 2). Youden plot shows random deviation. 
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Figure 4 XRF data after equipment overhaul (B values are 
1/3 those shown in figure 2) . Random deviation 
has been reduced leaving gross systematic deviation. 
Note large number of outliers in the T plot. 



Table 1 Summary of control data used to plot the figures 



B S A S B S T S D 



mg/1 solids (fig. 1) 2.6 5.0 10.8 9.9 19.7 6.2 



% Sulphur 

- raw data (fig. 2) .256 .672 .013 .031 .042 .023 



- before (fig. 3) .249 .220 .0092 .0094 .0167 .0083 o 



to 



- after (fig. 4) .263 .227 .0118 .0102 .0218 .0044 



Table 2 Calculated level of random vs systematic standard deviation 



within-run 

s_ 



systematic 

s 



between-run 

8 



mg/1 solids (fig. 1) 



4.4 



9.4 



10.3 



% Sulphur 

- raw data (fig. 2) 



- before (fig. 3) 



ana 



lysis not valid s, ^ s 



0059 



A r B 



.0073 



see text 



.0093 



H 

? 
H 



- after (fig. 4) 



0031 



.0106 



.0110 
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When consideration is taken of the field sampling 
variability and other effects, the between-run precision of 
± 4.4% in the range 0.25 to 0.75 %S is more than adequate. 
However, this evaluation suggests that the precision could be 
improved to + 1.5% if necessary by control of systematic error. 

•In this XRF study the A and B data was accumulated only, 
but it is obvious that, had the need existed, the A and B 
samples could have been used to control the instrumental 
calibration. It is this potential for control via daily analy- 
sis of the T and D plots that makes this technique most 
valuable. 
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APPENDIX SECTION I; ANALYTICAL REPRODUCIBILITY 
PART D : 



CALIBRATION CONTROL VIA THE 'A - B' TECHNIQUE 

The following discussion describes in more detail the 
application of the technique for detection of systematic 
error presented in Part C. Whereas Part C described the 
theory involved, this section discusses the practical 
factors to be considered by the analyst in choosing con- 
trol samples, and how to interpret the findings both graph- 
ically and numerically. Emphasis is placed on graphical 
interpretation, however, because usually the calculator/ 
plotter facilities referred to are not available, and in 
any event they would only be used periodically to sum- 
marize performance. The primary intent is the use of this 
technique as an on-line control tool, in which instance, 
mathematical analysis is unnecessary. 

The underlying assumption in any analytical pro- 
cedure is that sufficient development study has been 
undertaken to demonstrate and ensure that the response 
of the analytical technique and instrumentation, to the 
parameter of interest, follows certain physical and/or 
chemical laws in a relatively stable and predictable 
fashion. Sudden changes in the slope and intercept of 
the calibration curve are not to be expected so that 
when such are observed they are indicative of error in 
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preparation of the reference material used for calibra- 
tion, error in warm-up or operation of the instrument 
degradation in reagent quality or instrument response, 
incipient instrument failure, or poor instrument design. 

When any change in response is observed, a decision 
is required either to continue using a previous calibra- 
tion, or to reject it in favour of a new response/concen- 
tration curve. This decision can have one of four out- 
comes : 

a) to decide correctly that no change is required. 

b) to decide correctly that change is required. 

c) to decide incorrectly (Type I error) that 
change is required. 

d) to decide incorrectly (Type II error) that no 
change is required. 

Small changes in slope or intercept are inherent in 
any measuring system, and within limits must be ignored 
to avoid the possibility of a Type I error. 

Large changes in slope and/or intercept are not to 
be expected and, therefore, require positive confirma- 
tion of their validity. Under these circumstances there 
is little chance of a Type II error unless the analyst 
is asleep at the post, but Type I error is highly pro- 
bable if the observed change is not confirmed by an alter- 
nate independent check. 

Systematic error, therefore, is generally the re- 
sult of Type I error, i.e. the result of a decision to 
change the calibration based on insufficient evidence. 
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The technique described below is one source of additional 

information that can be used to assist in making a proper 
decision. 

The "A-B" Control Technique 

The "A-B" technique for intra- laboratory control of 
calibration (which is based on the Youden technique 
for detecting systematic error in evaluating inter-labora- 
tory compatability) ,, is outlined below. 

1) Two known (or unknown) standards (or stable 
natural samples) , falling within a single 
operating range for the methodology to be 
controlled, are each analyzed once per 
analytical run or batch of samples, immed- 
iately after the instrument has been completely 
calibrated for that run or batch. 

2) The two results A and B and their sum (A+B) 
and their difference (A-B) are recorded and 
a sequential plot of these values is main- 
tained over time. 

3) The standard deviation of accumulated (A-B) 
data from a previous set can be used to 
estimate and set control limits within which 
the individual points (A+B) and (A-B) are 
expected to fall. Thus, 



Warning limits are (A+B) + 2s_ and (A-B) + 2s_ 
Cont rol limits are (A+B) ± 3s„ and (A-B) ± 3s„ 

JJ D 

(If only chance is involved, warning limits about the 
average value will be exceeded by less than 5% of the 
data, whereas control limits will be exceeded by less 
than 0.2% of the data. Therefore, any such data is 
suggestive of either systematic or gross error.) 
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s_ = standard deviation of difference (A-B) 



Z(D 2 ) - (ZD) 2 /n 



(n - 1) 
given D * (A-B) 
and n - number of A,B data pairs 



Note that (A+B) and (A-B) are either the calculated 
average values when A and B are unknown, or the known 
true values if A and B are known standards. The work- 
ing control limits will be assigned for specific methods 
based on experience with the calculated values of s_ for 
that method . 

4) Any individual point (A+B) falling outside the 
control limits is indicative of significant 
systematic error in calibration. No concentra- 
tions may be calculated until the source of error 
has been identified and corrected. This will 
probably require preparation of a new indepen- 
dent standard. 

5) Any individual point (A-B) falling outside the 
control limits may be the result of gross 
random error in either A or B . If this is not 
the case, then such a point is strongly sugges- 
tive of significant degradation in the precision 
of the analytical process. Further information 
should be available from other control procedures 
to confirm this suggestion. 

In most cases the "A-B" control samples are carried 
through the entire analytical process. However, separate 
"A-B" control over instrument calibration may be essen- 
tial if systematic error is a special problem. 
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Application of "A-B" Control Samples 

This control technique can be used to monitor the 
total analytical system or any part of it. Although 
systematic error is more usually related to miscalibra- 
tion in the final stage of the analytical process, it 
can occur at any stage whenever an assumption is made 
affecting the accurate transfer or measurement of the 
analyte or sample of which it is a part, for example, 
gravimetric balance operation, recovery of analyte 
after sample pre- treatment , digestion, etc. 

As was discussed above, the primary use of the "A-B" 
technique is to detect and thereby prevent any signifi- 
cant systematic error in the final stage of the analyti- 
cal process i.e. calibration of instrumental equipment. 
This involves on-the-bench application by the technician 
performing the analyses under control chart conditions. 

There are three possible ways to select concentra- 
tions for the two control samples : 

I) Both samples at the low end of the operating 
range (eg below 10-20% of full scale) . 

II) Both samples at the upper end of the operating 
range (eg. approximately 75% of full scale). 

Ill) One sample at the low end and one at the upper 
end of the operating scale. 

Case I : At the low end of the operating range, normal 
changes in slope cannot be detected within the resolu- 
tion of the operating scale so that any significant 
changes detected must be attributed to error in the 
assessment of the blank or calibration intercept. 
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Case II ; Unless the blank is large and subject to 
significant variability, the larger proportion of any 
change detected can be attributed to error in deter- 
mining calibration slope. 

Case III : Ideally, control samples should be prepared 
to cover both Case I and Case II , but in actual 
practice this is not essential. Experience with a 
given analytical procedure will usually show that 
detected errors are more likely due to either slope 
or blank error but not both. Two controls separated 
in concentration provide, by their difference (A-B) , 
an estimate of the slope # which is accepted as long as 
it lies within the control limits imposed on (A-B) . 
If the slope is in control then an outlier in (A+B) is 
strong evidence of systematic error in the blank or 
intercept. If (A-B) is out of control it becomes more 
difficult to determine the source of error. Attention 
should then be paid to the individual values of A and B, 
keeping in mind the most likely source of problem for 
that analytical methodology, but also recalling that 
both blank and slope error are being monitored. 

When applying Case III it is essential to note that 
the standard deviations of A and B should be approximat- 
ely equal so that excessive separation in concentration 
must be avoided. The following criteria should be 
considered. 

a) both control samples must fall in the same 
operating scale. 

b) both controls should fall in an area of the 
calibration which is routinely expected to 
remain linear. 
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c) both controls should bracket the range on the 
scale within which most natural sample concen- 
trations are observed. 

d) they should probably not be separated by more 
than 50% (or less than 40%) of full scale 
unless it is known that the analytical standard 
deviation is independent of concentration as 
might be the case in titrimetric methods, for 
example. 

If the standard deviation in the operating range is 
exactly proportional to concentration and it is known 
that there is no significant blank or intercept, widely 
separated concentrations for the control samples can be 
used, but the data for the higher concentration control 
should be divided by the ratio between the high and 
low controls before plotting the (A+B) and (A-B) graphs. 
Thus if B=5A plot (A+B/5) and (A-B/5) . 

When plotting the "A-B" data care should be taken 
not to use too large a plotting scale for the concen- 
tration axis to avoid seeing insignificant changes. 
Only data falling outside the warning limits need be 
considered. Warning limits and control limits should 
be based upon previous calculations of s , but could 
be rounded up to the next closest unit used in reporting 
data and should not normally require adjustment. The 
usual 8" x 10" graph paper should be utilized for 
several months before a new graph is required. 

Preparation of "A-B" Control Samples 

The "A-B" control samples form an alternate indepen- 
dent check on calibration. They are re-analyzed day 
after day (or run after run) to provide a continuing 
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record that the routine calibration procedure has been 
properly completed and that no significant error in 
preparation of reagents, standards or equipment has 
occurred. In order to achieve this it is essential 
that: 

1) A new pair of "A-B" controls be prepared and 
analyzed for at least three days before the 
old set is exhausted. 

2) Sufficient control sample be prepared to last 
for at least 20 runs (approximately one month 
at one run/day), if not longer. 

3) Even if the concentration of the controls are 
not known, as is the case when natural samples 
are used for this purpose, there should be 
reason to expect that both A and B control 
samples will be stable for the required period 
of time (20 runs at least) . 

4) Preparation of new "stock standard" solutions 
used to prepare the "working standards" 
required for daily instrument calibration 
should be scheduled to occur in the middle 

of the period controlled by a particular set 
of "A-B" samples. 

UNDER NO CIRCUMSTANCES SHOULD STOCK STANDARDS 
AND CONTROLS BE CHANGED WITHIN LESS THAN THREE 
DAYS OF EACH OTHER. 

5) The "A-B" control samples, after about four 
analyses, can be considered as reference materi- 
als and should be treated as such. Care must be 
taken in storage and handling to ensure their 
concentration is not affected. (eg. insufficient 
mixing prior to sub-sampling) . 
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Calculation of Standard Deviations Using "A-B" Data 

1. A ■ (EA)/n = average of A's 
B = (EB)/n ■ average of B's 
T = A + B = average of sums (T = A+B) 
D = A - B = average of differences (D = A-B) 

2 . Calculate variances 



of sums: s 2 = (ET 2 - nT 2 ) / (n - 1) 

of differences: s 2 = (ED 2 - nD 2 ) / ( n ■ 1) 

of results A: s 2 = (EA 2 - nA 2 ) / (n - 1) 

of results B: s 2 = (EB 2 - nB 2 ) / (n - 1) 

D 



3. Calculate Total variance (between-run) 



s 2 = (s A 2 + s B 2 )/2 
= (s T 2 + s D 2 )/4 

" (S T 2 + S D 2 " S A 2 ' S B 2) / 2 



4 . Calculate Random variance (within-run) 

s 2 = s n 2 /2 

w D 

5 . Calculate Systematic component of Total Variance 



s e 2 = (s 2 -s w 2 ) . (s T 2 - s A 2 - s B 2 )/2 



6. Calculate Standard Deviations: s, s w , s 



7 . Calculate Warning limits for T and D 

less than 5% chance that (T - T) or (D-D) 
will exceed ± 2s n if systematic error is 
controlled. 
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8. Calculate Control Limits for T and D 

less than 0.2% chance that (T - T) or (D - D) will 
exceed ± 3s if systematic error is controlled. 

When "A-B" Data is Out of Control 

In order to assist in determining the possible cause 
for (A+B) or (A-B) data falling outside the assigned con- 
trol limits, certain additional information should be 
documented and plotted and readily available, as necessary: 

1) Daily plot showing the instrumental response in 
physical units (eg. ml, mg, absorbance, %T, radi- 
ation counts, scale reading, etc.) to a specific 
independent high level standard, under specified 
fixed conditions of instrument operation and 
instrument control settings including range 
expansion. 

2) Daily plot showing the amount of reagent or other 
background normally zeroed out by routine instru- 
ment operating procedures (eg. in color imetric 
analysis the reagent blank relative to distilled 
water or other pure solvent) . 

3) Daily plot showing the level of blank introduced 
during sample pre-treatment procedures exclusive 
of the background noted above (eg. digestion 
blank) whether or not it is zeroed out by instru- 
ment adjustment or by calculation. 

4) When continuously variable range expansion or 
scale adjustment is used to convert the response 
directly to concentration units, a daily plot of 
the control setting used must be maintained. If 
the instrument control is not presently calibrated 
to indicate the setting, it must be replaced by a 
ten- turn, or other suitable, control knob to per- 
mit this. 
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Limits may be set for the above data, if desired, 
as additional control against deterioration even when 
"A-B" data indicates the system is under control. 

In most cases, examination of such data plots will 
provide the clue required and point to a means for 
correcting the source of error whether due to 

a) deteriorating instrument response requiring 
eventual instrument overhaul 

b) deteriorating or contaminated analytical reagents 

c) poor technique in sample preparation or final 
analysis (poor reproducibility) . 

d) poor recovery or unsatisfactory calibration 
(poor accuracy) . 

Conclusion 

Poor calibration control is often tolerated on the 
grounds that lack of sample homogeniety limits analytical 
precision. It must be kept in mind that poor calibration 
affects accuracy not precision and that even small biases 
can be corrected in spite of imprecision. 



APPENDIX SECTION I: ANALYTICAL REPRODUCIBILITY 
PART E; 

EVALUATION OF ANALYTICAL METHODS BY A-B TECHNIQUE 

The previous part discussed the A-B technique as 
an on-line tool to prevent systematic error. In this 
chapter the data produced is used at a later date to 
characterize and evaluate the method itself. 

At regular intervals A-B control data is retrieved 
from the technician and fed into a desk-top calculator/ 
plotter to obtain the output shown in the figures fol- 
lowing. Of particular interest is the plot of B versus 
A in which the points are joined sequentially. Lines 
tend to run from lower left to upper right and their 
predominance in almost all such diagrams verifies the 
difficulties in complete elimination of systematic 
error between analytical calibrations. This diagram 
immediately characterizes the analytical system. 

The scale used to plot these figures is chosen so 
that one scale division is equal to or better than the 
expected standard deviation of the test. If the 
system is OK the diagram tends to vanish, i.e. all 
points fall within ± 3 divisions of the averages. As 
the system deteriorates, or if it is worse than 
expected the various figures become more visible. 

The two circular rings (bulls-eyes) are centred 
on the averages A and B (or alternatively their known 
values) . The inner ring has a radius 2s w and the 
points should be uniformly scattered in all four quad- 
rants. The outer ring becomes larger as systematic 
error becomes more significant (less controlled) and 
has a radius of 2s. 
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The available data is also used to plot warning 
and control limit lines for T (=A+B) and D (=A-B) . 
These lines are based on + 2s_ and ± 3s respectively 
about both T and D. These calculated limits can be 
compared to the working values preassigned to the 
specific parameter. 

The plots of A, B, T and D versus time relative 
to their respective averages help to determine whether 
the controls are effective. Some systems are more 
effectively controlled and/or controllable than others. 
The diagrams help to identify problems quickly. The 
following figures demonstrate this for typical systems. 

The validity of the calculated standard deviations 
depends greatly upon the similarity of s and s and the 
stability of the controls . Until and unless calibration 
slope is controlled between runs, the calculated values 
s and s w should be interpreted with care since they will 
be too dependent on the standard deviation of the high 
control sample. 

Poor calibration control is often tolerated on 
the grounds that lack of sample homogeniety limits 
analytical precision. It must be kept in mind that 
poor calibration affects accuracy not precision and 
that even small biases can be corrected in spite of 
imprecision. 
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Discussion of Examples 

In the following examples the nominal values rather 
than the calculated averages were used as the reference 
points for plotting A, B, A + B and A - B versus time, 
and the 'bulls-eyes' and control limits are centred on 
this point. Displacement of the data suggests bias only 
if the nominal values are accurate. 



Total Iron: 



Evaluation: 



- manual digestion followed by Technicon 
AAII analysis 

- while readings can be made to nearest 
0.01 mg/1 this is not usually required 
since data is reported only to nearest 
0.05 mg/1 (in 1975) 

- precision is being limited by non-standardized 
reading practices 

- deviations from day to day are predominantly 
random 

- randomness in digestion step is probably 
obscuring systematic error in calibration 

Reactive Silicates : - direct analysis by Technicon 
AAII (0-2 mg/1) 

- readings made to at least nearest 0.02 mg/1 



Evaluation: 



I 
I 
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- gross systematic error in slope control 
from day to day was brought to attention 
of staff part way through the period. It 
was found that the Std./Cal. control was 
being reset daily. 

- latter part of data shows vast improvement 
in precision of A attained by leaving the 
Std./Cal. control alone 

- in this example control limits should be 
based on the std. dev. of the B control 
sample 

- drift in control A?, or else change in 
sensitivity not compensated because of 
requirement that Std./Cal. control remain 
fixed during latter period? 

Total Kjeldahl Nitrogen (1974) : - manual persulphate 

digestion followed by Technicon II analysis 
for ammonia (0-2. mg/1) 

- readings made to nearest 0.01 and reported 
to nearest 0.05 if greater than 1.0 mg/1 



Evaluation: 



random effects during digestion step are 
obscuring systematic error shown by trends 
in A and B. 

within- run std. dev. overestimated because 
of excessive variation in the A control, 
neither precision or bias are well controlled 
during this period. 



I 
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Total Kjeldahl Nitrogen (1975) ; - as above 

Evaluation : - significant improvement in precision control 

both within and between runs 

- note std. dev. is relatively independant of 
range 

- this performance is probably the best to be 
expected, but it was confirmed by std. dev. 
calculated from routine in-run duplicates 

Typical Patterns 

B vs A A (low) B(high) A+B A-B 

1) Std. dev. limited by readability and constant throughout 
the operating scale, no systematic error. 



2) Std. dev. limited by readability, increases slightly with 
concentration through the range, no systematic error 



V 



3) Isolated gross error; other conditions as in 1) 

mm XL iW* f*f 

4) Isolated systematic error; other conditions as in 1) 

iff *L« ******* JUm* wv*wwv\ 



5) Difference between old and new batch of calibration 
standard 



7) Drift in calibration blank; slope under control 

^ mS\* «r**W S*/^ 
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8) Drift in calibration slope; precision as in 1) control 



limits will be two wide. 



I 



A A 



twivii*m 



9) Std. deviation worse than expected based on readability; 
probably limited by sample preparation (eg. digestion) ; 
system under control 




10) Change in within-run std.dev.; system under control 



& #~ IV- 



\\n 



wm 



11) Std.dev. worse than expected based on readability; 
systematic error masking expected precision (see 9) 



12) Data being "cooked' 



# 



wn»n 
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13) Systematic error; data being 'cooked'; calibration drift 



4JT /''¥ $d* mm 



14) Imprecision at upper end not due to systematic error 



PW 



15) Imprecision at upper end partially due to systematic 
error in blank 




um 



W[\\UW 



16) Imprecision at upper end partially due to systematic 
error in slope 
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Inner circle = 2 x std. dev'n within-run data 1^^- Q.OS\ 
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Overall std. dev'n for data from different runs 
Std. dev'n for data from the same run 



Controllable contribution to overall std. dev'n 
Inner circle = 2 x std. dev'n within-run data 
Outer circle = 2 x std. dev'n between run data 
Concentration per division on the diagram 
Calculated Warning limit for T and D 
Calculated Control limit for T and D 
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A vs. B DATA REPORT 



Ontario 



Parameter *To~T ft U K 3 e l"fr f^\\ \- ^ C^«^w.»i, A»V»v - A ATT~ . 

Period Ruftu«>T - Ot.-rofee.tf- l*H4- 

A average 1 ■ (oL~±- j*J[A (r\fc«nl*v«4_ \4>o) std. dev'n OQ9S"? 
B average 0%1^ **a\\ (y\o^>Aaj 0-So std. dev'n 0-0S33 

Overall std. dev'n for data from different runs Q. c~1<) 

Std. dev'n for data from the same run O*0S9 



Controllable contribution to overall std. dev'n Q. 063 

Inner circle = 2 x std. dev'n within-run data Q-llft 

Outer circle = 2 x std. dev'n between run data O- 1£^ 



oo Concentration per division on the diagram 
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m Calculated Warning limit for T and D 
Calculated Control limit for T and D 
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A vs. B DATA REPORT 



Ontario 

Parameter T&TAL K3fcLT>fim, tvj (tW>Jt JU«'r> - AATl) Laborator 

Period Ociofteft. - VOoi6.etgtfe.A- 1975 

A average l-C>£^> **\jI CaowiaoI Lt>o) std. dev'n Q. 03.74 
B average 0.791 ^ji (a6^ja«J Oto) std. dev'n O-QX^I 

Overall std. dev'n for data from different runs Q.Q3.S 

Std. dev'n for data from the same run 0.ox\ 



Controllable contribution to overall std. dev'n 
Inner circle = 2 x std. dev'n within-run data 
Outer circle = 2 x std. dev'n between run data 
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cr, Concentration per division on the diagram 
h Calculated Warning limit for T and D 
Calculated Control limit for T and D 
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Evaluation of Standard Deviations 

Experience has shown that precision is not being 
limited by analytical readability if the in-run standard 
deviation is greater than about 1.5X the value 'results 
read to nearest' . When these two numbers are about 
equal it can be implied that results can be reported 
better if the need exists. It is occasionally observed 
that two laboratories using identical methodology but 
different reporting practices will report results in 
one case to the nearest 0.2 and find a standard devia- 
tion of 0.3 to 0.5 units while the other reports to 
the nearest 1. and calculates a standard deviation of 
0.2 or better. In the latter case the apparent repro- 
ducibility is an artifact of poor reporting practices. 

Comparison of s „ , s_ , s.,, s and s, . can reveal 
r A B Id md hd 

typical performance patterns. If A < B then; 



If 



S A * S B 



A B 
and S B > S hd 



slope control is required and will be effective 
in improving between-run precision at high 
concentration levels. Interpretation of s and 
s wiU be inappropriate until control is 

enforced. 



If 



S A . S B 



but S B - S hd, 



imprecision at high concentration levels is 
related more to sample aliquoting difficulty. 
Slope control will help prevent bias but will 
not improve precision. Interpretation of s 
and s is improper. 



If S A - S B 
and s > s 



and s. 



w 



'Id. 



If S A * S B 

and s > s.. 
w 

but s w < s ld 



blank or baseline control will be effective in 
improving between-run precision to the level 
permitted by reproducability of sample aliquot- 
ting or analytical technique. 

blank or baseline control will prevent bias but 
precision is limited by the nature of the sample 
or analytical technique. 
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